2026-10-29 19:40:59,163 [ 187052 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-10-29 19:40:59,163 [ 187052 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-10-29 19:40:59,163 [ 187052 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-10-29 19:40:59,163 [ 187052 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_m6exb1 --privileged --dns-search='.' --memory=30703235072 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel1_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_distributed_ddl/test.py::test_kill_query[configs_secure]' 'test_distributed_ddl/test.py::test_macro[configs]' 'test_distributed_ddl/test.py::test_macro[configs_secure]' 'test_distributed_ddl/test.py::test_on_server_fail[configs]' 'test_distributed_ddl/test.py::test_on_server_fail[configs_secure]' 'test_distributed_ddl/test.py::test_optimize_query[configs]' 'test_distributed_ddl/test.py::test_optimize_query[configs_secure]' 'test_distributed_ddl/test.py::test_rename[configs]' 'test_distributed_ddl/test.py::test_rename[configs_secure]' 'test_distributed_ddl/test.py::test_replicated_without_arguments[configs]' 'test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure]' 'test_distributed_ddl/test.py::test_simple_alters[configs]' 'test_distributed_ddl/test.py::test_simple_alters[configs_secure]' 'test_distributed_ddl/test.py::test_socket_timeout[configs]' 'test_distributed_ddl/test.py::test_socket_timeout[configs_secure]' 'test_failed_mutations/test.py::test_backoff_clickhouse_restart[False]' 'test_failed_mutations/test.py::test_backoff_clickhouse_restart[True]' test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table test_failed_mutations/test.py::test_exponential_backoff_setting_override 'test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True]' 'test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False]' test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree 'test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False]' 'test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True]' test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table test_keeper_profiler/test.py::test_profiler test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_snapshots/test.py::test_ephemeral_after_restart test_keeper_snapshots/test.py::test_invalid_snapshot test_keeper_snapshots/test.py::test_snapshot_size test_keeper_snapshots/test.py::test_state_after_restart test_library_bridge/test_exiled.py::test_bridge_dies_with_parent test_limit_materialized_view_count/test.py::test_limit_materialized_view_count 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_log_family_s3/test.py::test_stripe_log_truncate test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user]' test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete test_merge_tree_empty_parts/test.py::test_empty_parts_summing 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_move_ttl_broken_compatibility/test.py::test_bc_compatibility test_mutations_hardlinks/test.py::test_delete_and_drop_mutation test_mutations_hardlinks/test.py::test_drop_mutation test_mutations_hardlinks/test.py::test_modify_mutation test_mutations_hardlinks/test.py::test_update_mutation test_mysql_protocol/test.py::test_golang_client test_mysql_protocol/test.py::test_java_client_binary test_mysql_protocol/test.py::test_java_client_text test_mysql_protocol/test.py::test_mysql_affected_rows test_mysql_protocol/test.py::test_mysql_boolean_format test_mysql_protocol/test.py::test_mysql_client test_mysql_protocol/test.py::test_mysql_client_exception test_mysql_protocol/test.py::test_mysql_explain test_mysql_protocol/test.py::test_mysql_federated test_mysql_protocol/test.py::test_mysql_replacement_query test_mysql_protocol/test.py::test_mysql_select_user test_mysql_protocol/test.py::test_mysql_set_variables test_mysql_protocol/test.py::test_mysqljs_client test_mysql_protocol/test.py::test_php_client test_mysql_protocol/test.py::test_python_client test_mysql_protocol/test.py::test_types 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_distributed_ddl/test.py::test_kill_query[configs_secure] test_keeper_snapshots/test.py::test_ephemeral_after_restart test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] test_mysql_protocol/test.py::test_golang_client test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_log_family_s3/test.py::test_log_family_s3[Log] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] [gw5] [ 1%] PASSED test_keeper_snapshots/test.py::test_ephemeral_after_restart test_keeper_snapshots/test.py::test_invalid_snapshot [gw7] [ 2%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] test_log_family_s3/test.py::test_log_family_s3[StripeLog] [gw2] [ 3%] PASSED test_distributed_ddl/test.py::test_kill_query[configs_secure] test_distributed_ddl/test.py::test_macro[configs_secure] [gw1] [ 4%] PASSED test_mysql_protocol/test.py::test_golang_client test_mysql_protocol/test.py::test_java_client_binary [gw7] [ 5%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw9] [ 6%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] [gw7] [ 7%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] test_log_family_s3/test.py::test_stripe_log_truncate [gw2] [ 8%] PASSED test_distributed_ddl/test.py::test_macro[configs_secure] test_distributed_ddl/test.py::test_on_server_fail[configs_secure] [gw7] [ 9%] PASSED test_log_family_s3/test.py::test_stripe_log_truncate [gw0] [ 10%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] [gw1] [ 11%] PASSED test_mysql_protocol/test.py::test_java_client_binary test_mysql_protocol/test.py::test_java_client_text [gw9] [ 12%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] [gw3] [ 13%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] [gw1] [ 14%] PASSED test_mysql_protocol/test.py::test_java_client_text test_mysql_protocol/test.py::test_mysql_affected_rows [gw1] [ 15%] PASSED test_mysql_protocol/test.py::test_mysql_affected_rows test_mysql_protocol/test.py::test_mysql_boolean_format [gw1] [ 16%] PASSED test_mysql_protocol/test.py::test_mysql_boolean_format test_mysql_protocol/test.py::test_mysql_client [gw0] [ 17%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] [gw1] [ 18%] PASSED test_mysql_protocol/test.py::test_mysql_client test_mysql_protocol/test.py::test_mysql_client_exception [gw1] [ 19%] PASSED test_mysql_protocol/test.py::test_mysql_client_exception test_mysql_protocol/test.py::test_mysql_explain [gw1] [ 20%] PASSED test_mysql_protocol/test.py::test_mysql_explain test_mysql_protocol/test.py::test_mysql_federated [gw5] [ 21%] PASSED test_keeper_snapshots/test.py::test_invalid_snapshot [gw9] [ 22%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] test_keeper_snapshots/test.py::test_snapshot_size [gw4] [ 23%] PASSED test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] [gw3] [ 24%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] [gw5] [ 25%] PASSED test_keeper_snapshots/test.py::test_snapshot_size test_keeper_snapshots/test.py::test_state_after_restart [gw0] [ 26%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] [gw9] [ 27%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] [gw5] [ 28%] PASSED test_keeper_snapshots/test.py::test_state_after_restart [gw2] [ 29%] PASSED test_distributed_ddl/test.py::test_on_server_fail[configs_secure] test_distributed_ddl/test.py::test_optimize_query[configs_secure] test_mutations_hardlinks/test.py::test_delete_and_drop_mutation [gw3] [ 30%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant [gw2] [ 31%] PASSED test_distributed_ddl/test.py::test_optimize_query[configs_secure] test_distributed_ddl/test.py::test_rename[configs_secure] [gw0] [ 32%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] [gw3] [ 33%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] [gw0] [ 34%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] [gw4] [ 35%] PASSED test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table [gw4] [ 36%] PASSED test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table test_failed_mutations/test.py::test_exponential_backoff_setting_override [gw4] [ 37%] PASSED test_failed_mutations/test.py::test_exponential_backoff_setting_override test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] [gw6] [ 38%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete [gw0] [ 39%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] [gw5] [ 40%] PASSED test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant [gw2] [ 41%] PASSED test_distributed_ddl/test.py::test_rename[configs_secure] test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] [gw3] [ 42%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table [gw0] [ 43%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] [gw4] [ 44%] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] [gw2] [ 45%] PASSED test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] test_distributed_ddl/test.py::test_simple_alters[configs_secure] [gw4] [ 46%] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree [gw0] [ 47%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] [gw3] [ 48%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] [gw4] [ 49%] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] [gw0] [ 50%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] [gw5] [ 51%] PASSED test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table [gw9] [ 52%] PASSED test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete test_merge_tree_empty_parts/test.py::test_empty_parts_summing [gw2] [ 53%] PASSED test_distributed_ddl/test.py::test_simple_alters[configs_secure] test_distributed_ddl/test.py::test_socket_timeout[configs_secure] [gw9] [ 54%] PASSED test_merge_tree_empty_parts/test.py::test_empty_parts_summing [gw3] [ 55%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] [gw0] [ 56%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] [gw4] [ 57%] PASSED test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] test_keeper_profiler/test.py::test_profiler test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader [gw0] [ 58%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] [gw3] [ 59%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] [gw4] [ 60%] PASSED test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] [gw6] [ 61%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] [gw0] [ 62%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] test_limit_materialized_view_count/test.py::test_limit_materialized_view_count [gw6] [ 63%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_library_bridge/test_exiled.py::test_bridge_dies_with_parent [gw0] [ 64%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] [gw5] [ 65%] PASSED test_keeper_profiler/test.py::test_profiler [gw6] [ 66%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] [gw0] [ 67%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] [gw0] [ 68%] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] [gw2] [ 69%] PASSED test_distributed_ddl/test.py::test_socket_timeout[configs_secure] test_distributed_ddl/test.py::test_macro[configs] [gw9] [ 70%] FAILED test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader [gw3] [ 71%] SKIPPED test_library_bridge/test_exiled.py::test_bridge_dies_with_parent test_move_ttl_broken_compatibility/test.py::test_bc_compatibility [gw4] [ 72%] PASSED test_limit_materialized_view_count/test.py::test_limit_materialized_view_count test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 [gw7] [ 73%] PASSED test_mutations_hardlinks/test.py::test_delete_and_drop_mutation test_mutations_hardlinks/test.py::test_drop_mutation [gw7] [ 74%] PASSED test_mutations_hardlinks/test.py::test_drop_mutation test_mutations_hardlinks/test.py::test_modify_mutation [gw7] [ 75%] PASSED test_mutations_hardlinks/test.py::test_modify_mutation test_mutations_hardlinks/test.py::test_update_mutation test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw7] [ 76%] PASSED test_mutations_hardlinks/test.py::test_update_mutation [gw6] [ 77%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw5] [ 78%] PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 [gw9] [ 79%] PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw3] [ 80%] PASSED test_move_ttl_broken_compatibility/test.py::test_bc_compatibility [gw1] [ 81%] PASSED test_mysql_protocol/test.py::test_mysql_federated test_mysql_protocol/test.py::test_mysql_replacement_query [gw1] [ 82%] PASSED test_mysql_protocol/test.py::test_mysql_replacement_query test_mysql_protocol/test.py::test_mysql_select_user [gw1] [ 83%] PASSED test_mysql_protocol/test.py::test_mysql_select_user test_mysql_protocol/test.py::test_mysql_set_variables [gw1] [ 84%] PASSED test_mysql_protocol/test.py::test_mysql_set_variables test_mysql_protocol/test.py::test_mysqljs_client [gw2] [ 85%] PASSED test_distributed_ddl/test.py::test_macro[configs] test_distributed_ddl/test.py::test_on_server_fail[configs] [gw1] [ 86%] PASSED test_mysql_protocol/test.py::test_mysqljs_client test_mysql_protocol/test.py::test_php_client [gw1] [ 87%] PASSED test_mysql_protocol/test.py::test_php_client test_mysql_protocol/test.py::test_python_client [gw1] [ 88%] PASSED test_mysql_protocol/test.py::test_python_client test_mysql_protocol/test.py::test_types [gw1] [ 89%] PASSED test_mysql_protocol/test.py::test_types [gw6] [ 90%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw2] [ 91%] PASSED test_distributed_ddl/test.py::test_on_server_fail[configs] test_distributed_ddl/test.py::test_optimize_query[configs] [gw2] [ 92%] PASSED test_distributed_ddl/test.py::test_optimize_query[configs] test_distributed_ddl/test.py::test_rename[configs] [gw2] [ 93%] PASSED test_distributed_ddl/test.py::test_rename[configs] test_distributed_ddl/test.py::test_replicated_without_arguments[configs] [gw2] [ 94%] PASSED test_distributed_ddl/test.py::test_replicated_without_arguments[configs] test_distributed_ddl/test.py::test_simple_alters[configs] [gw2] [ 95%] PASSED test_distributed_ddl/test.py::test_simple_alters[configs] test_distributed_ddl/test.py::test_socket_timeout[configs] [gw8] [ 96%] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] [gw2] [ 97%] PASSED test_distributed_ddl/test.py::test_socket_timeout[configs] [gw8] [ 98%] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] [gw8] [ 99%] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] [gw8] [100%] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] =================================== FAILURES =================================== ______________________ test_reconfig_remove_2_and_leader _______________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_reconfig_remove_2_and_leader(started_cluster): """ Remove 2 followers from a cluster of 5. Remove leader from 3 nodes. """ global zk1, zk2, zk3, zk4, zk5 zk1 = create_client(node1) config = ku.get_config_str(zk1) print("Initial config", config) assert len(config.split("\n")) == 5 for i in range(100): zk1.create(f"/test_two_{i}", "somedata") zk4 = create_client(node4) zk4.sync("/test_two_0") ku.wait_configs_equal(config, zk4) zk5 = create_client(node5) zk5.sync("/test_two_0") ku.wait_configs_equal(config, zk5) for i in range(100): assert zk4.exists(f"test_two_{i}") assert zk5.exists(f"test_two_{i}") zk4.create(f"/test_two_{100 + i}", "otherdata") zk2 = create_client(node2) config = zk2.reconfig(joining=None, leaving="4,5", new_members=None) print("After removing 4,5", config) assert len(config.split("\n")) == 3 assert "node1" in config assert "node2" in config assert "node3" in config assert "node4" not in config assert "node5" not in config zk1.stop() zk1 = create_client(node1) zk1.sync("/test_two_0") ku.wait_configs_equal(config, zk1) for i in range(200): assert zk1.exists(f"test_two_{i}") assert zk2.exists(f"test_two_{i}") assert not node1.contains_in_log(log_msg_removed) assert not node2.contains_in_log(log_msg_removed) assert not node3.contains_in_log(log_msg_removed) assert node4.contains_in_log(log_msg_removed) assert node5.contains_in_log(log_msg_removed) assert ku.is_leader(cluster, node1) for i in range(100): zk1.create(f"/test_leader_{i}", "somedata") # when a leader gets a remove request, it must yield leadership config = zk1.reconfig(joining=None, leaving="1", new_members=None) print("After removing 1 (leader)", config) assert len(config.split("\n")) == 2 assert "node1" not in config assert "node2" in config assert "node3" in config assert "node4" not in config assert "node5" not in config zk2.stop() zk2 = create_client(node2) zk2.sync("/test_leader_0") > ku.wait_configs_equal(config, zk2) test_keeper_reconfig_remove_many/test.py:119: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/keeper_utils.py:413: in wait_configs_equal right_config = sorted(get_config_str(right_zk).split("\n")) helpers/keeper_utils.py:402: in get_config_str return zk.get("/keeper/config") helpers/keeper_utils.py:178: in get return self.execute_query(f"get '{path}'", timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = query = "get '/keeper/config'", timeout = 60.0 def execute_query(self, query: str, timeout: float = 60.0) -> str: output = io.BytesIO() self.proc.stdin.write(query.encode() + b"\n") self.proc.stdin.flush() events = self.poller.poll(timeout) if not events: raise TimeoutError(f"Keeper client returned no output") for fd_num, event in events: if event & (select.EPOLLIN | select.EPOLLPRI): file = self._fd_nums[fd_num] if file == self.proc.stdout: while True: chunk = file.readline() if chunk.endswith(self.SEPARATOR): break output.write(chunk) elif file == self.proc.stderr: self.proc.stdout.readline() > raise KeeperException(self.proc.stderr.readline().strip().decode()) E helpers.keeper_utils.KeeperException: Code: 999. Coordination::Exception: Received error in heartbeat response: Operation timeout. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): helpers/keeper_utils.py:160: KeeperException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-10-29 19:43:03.584000 [ 725 ] INFO : Running tests in /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/test.py (cluster.py:2738, start) 2026-10-29 19:43:03.585000 [ 725 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 19:43:03.601000 [ 725 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany-gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:43:03.620000 [ 725 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:43:03.637000 [ 725 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany-gw9 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:43:03.637000 [ 725 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:43:03.659000 [ 725 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany-gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:43:03.683000 [ 725 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:43:03.706000 [ 725 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany-gw9 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:43:03.706000 [ 725 ] DEBUG : Command:[docker container list --all --filter name='^/roottestkeeperreconfigremovemany-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:43:03.725000 [ 725 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:43:03.725000 [ 725 ] DEBUG : No running containers for project: roottestkeeperreconfigremovemany-gw9 (cluster.py:879, cleanup) 2026-10-29 19:43:03.725000 [ 725 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:43:03.741000 [ 725 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:43:03.742000 [ 725 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:43:03.766000 [ 725 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:43:03.766000 [ 725 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:43:03.766000 [ 725 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:43:03.766000 [ 725 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:43:03.789000 [ 725 ] DEBUG : Stdout:2 (cluster.py:145, run_and_check) 2026-10-29 19:43:03.789000 [ 725 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:43:03.808000 [ 725 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:43:03.808000 [ 725 ] DEBUG : Volumes pruned: 2 (cluster.py:915, cleanup) 2026-10-29 19:43:03.808000 [ 725 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-10-29 19:43:03.809000 [ 725 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:43:03.809000 [ 725 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:43:03.809000 [ 725 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:43:03.810000 [ 725 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:43:03.811000 [ 725 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper1.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:43:03.812000 [ 725 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/database (cluster.py:4758, create_dir) 2026-10-29 19:43:03.812000 [ 725 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/logs (cluster.py:4769, create_dir) 2026-10-29 19:43:03.812000 [ 725 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:43:03.812000 [ 725 ] DEBUG : Setup directory for instance: node2 (cluster.py:2758, start) 2026-10-29 19:43:03.813000 [ 725 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:43:03.813000 [ 725 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:43:03.813000 [ 725 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:43:03.814000 [ 725 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:43:03.814000 [ 725 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper2.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:43:03.814000 [ 725 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/database (cluster.py:4758, create_dir) 2026-10-29 19:43:03.814000 [ 725 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/logs (cluster.py:4769, create_dir) 2026-10-29 19:43:03.815000 [ 725 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:43:03.815000 [ 725 ] DEBUG : Setup directory for instance: node3 (cluster.py:2758, start) 2026-10-29 19:43:03.815000 [ 725 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:43:03.815000 [ 725 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:43:03.815000 [ 725 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper3.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/database (cluster.py:4758, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/logs (cluster.py:4769, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:43:03.816000 [ 725 ] DEBUG : Setup directory for instance: node4 (cluster.py:2758, start) 2026-10-29 19:43:03.817000 [ 725 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:43:03.817000 [ 725 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:43:03.817000 [ 725 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:43:03.817000 [ 725 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:43:03.817000 [ 725 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper4.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/database (cluster.py:4758, create_dir) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/logs (cluster.py:4769, create_dir) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Setup directory for instance: node5 (cluster.py:2758, start) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:43:03.818000 [ 725 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper5.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/database (cluster.py:4758, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/logs (cluster.py:4769, create_dir) 2026-10-29 19:43:03.819000 [ 725 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:43:03.820000 [ 725 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env (cluster.py:96, _create_env_file) 2026-10-29 19:43:03.821000 [ 725 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:43:03.821000 [ 725 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:43:03.821000 [ 725 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:43:03.821000 [ 725 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:43:03.832000 [ 725 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 19:43:03.833000 [ 725 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env --project-name roottestkeeperreconfigremovemany-gw9 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node4 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node5 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node1 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by node3 (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node3 Pulling (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Stderr: node3 Pulled (cluster.py:147, run_and_check) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env --project-name roottestkeeperreconfigremovemany-gw9 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-10-29 19:43:12.142000 [ 725 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env --project-name roottestkeeperreconfigremovemany-gw9 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Network roottestkeeperreconfigremovemany-gw9_default Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Network roottestkeeperreconfigremovemany-gw9_default Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-10-29 19:43:15.643000 [ 725 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:15.646000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.648000 [ 725 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:43:15.651000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.651000 [ 725 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.8.6... (cluster.py:3155, start) 2026-10-29 19:43:15.655000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.657000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.759000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.862000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:15.964000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.066000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.169000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.271000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.374000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.483000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.586000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.688000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.790000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.892000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:16.995000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.097000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.200000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.302000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.405000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.508000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.610000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.713000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.815000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:17.918000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.021000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.124000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.228000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/20edf75d366089140228c51198be2bed183a7e487847ff36a8d91d2842669d54/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.229000 [ 725 ] DEBUG : ClickHouse node1 started (cluster.py:3159, start) 2026-10-29 19:43:18.229000 [ 725 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.230000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.231000 [ 725 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:43:18.232000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.232000 [ 725 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.8.3... (cluster.py:3155, start) 2026-10-29 19:43:18.233000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.235000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/2110fc3f934b9c995520e95c3470c95f8ef6a1457140f2a6b16735581621200c/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.235000 [ 725 ] DEBUG : ClickHouse node2 started (cluster.py:3159, start) 2026-10-29 19:43:18.235000 [ 725 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.236000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.237000 [ 725 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:43:18.237000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.238000 [ 725 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.8.5... (cluster.py:3155, start) 2026-10-29 19:43:18.239000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.241000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/a556679df800e26ed50af2417e624a557cf8d096a449dca81449276a173ed9e6/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.355000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/a556679df800e26ed50af2417e624a557cf8d096a449dca81449276a173ed9e6/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.356000 [ 725 ] DEBUG : ClickHouse node3 started (cluster.py:3159, start) 2026-10-29 19:43:18.359000 [ 725 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.360000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.361000 [ 725 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:43:18.362000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.362000 [ 725 ] DEBUG : Waiting for ClickHouse start in node4, ip: 172.16.8.2... (cluster.py:3155, start) 2026-10-29 19:43:18.363000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.369000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/638d1c415c0ac543525e334ab99725794dfa8ee3591ec592061345be56583e01/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.370000 [ 725 ] DEBUG : ClickHouse node4 started (cluster.py:3159, start) 2026-10-29 19:43:18.370000 [ 725 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.371000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.371000 [ 725 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:43:18.372000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.373000 [ 725 ] DEBUG : Waiting for ClickHouse start in node5, ip: 172.16.8.4... (cluster.py:3155, start) 2026-10-29 19:43:18.375000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.377000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/14598772c7bb4db5d46a7c578cec31c395159448419085a5fcf14283c33f0ca1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.378000 [ 725 ] DEBUG : ClickHouse node5 started (cluster.py:3159, start) ----------------------------- Captured stdout call ----------------------------- Initial config server.1=node1:9234;participant;1 server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 server.4=node4:9234;participant;1 server.5=node5:9234;participant;1 After removing 4,5 server.1=node1:9234;participant;1 server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 After removing 1 (leader) server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:43:18.379000 [ 725 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.381000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:18.896000 [ 725 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:18.899000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:19.030000 [ 725 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:19.032000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:19.764000 [ 725 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:19.766000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:19.910000 [ 725 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:19.912000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:20.136000 [ 725 ] DEBUG : run container_id:roottestkeeperreconfigremovemany-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2051, exec_in_container) 2026-10-29 19:43:20.137000 [ 725 ] DEBUG : Command:[docker exec roottestkeeperreconfigremovemany-gw9-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:43:20.192000 [ 725 ] DEBUG : run container_id:roottestkeeperreconfigremovemany-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2051, exec_in_container) 2026-10-29 19:43:20.192000 [ 725 ] DEBUG : Command:[docker exec roottestkeeperreconfigremovemany-gw9-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:43:20.252000 [ 725 ] DEBUG : run container_id:roottestkeeperreconfigremovemany-gw9-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2051, exec_in_container) 2026-10-29 19:43:20.253000 [ 725 ] DEBUG : Command:[docker exec roottestkeeperreconfigremovemany-gw9-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:43:20.311000 [ 725 ] DEBUG : run container_id:roottestkeeperreconfigremovemany-gw9-node4-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2051, exec_in_container) 2026-10-29 19:43:20.311000 [ 725 ] DEBUG : Command:[docker exec roottestkeeperreconfigremovemany-gw9-node4-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:43:20.367000 [ 725 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2026.10.29 19:43:19.864708 [ 601 ] {} RaftInstance: this server (4) has been removed from the cluster, will step down itself soon. config log idx 250 (cluster.py:145, run_and_check) 2026-10-29 19:43:20.368000 [ 725 ] DEBUG : run container_id:roottestkeeperreconfigremovemany-gw9-node5-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2051, exec_in_container) 2026-10-29 19:43:20.368000 [ 725 ] DEBUG : Command:[docker exec roottestkeeperreconfigremovemany-gw9-node5-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:43:20.423000 [ 725 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2026.10.29 19:43:19.915840 [ 600 ] {} RaftInstance: this server (5) has been removed from the cluster, will step down itself soon. config log idx 252 (cluster.py:145, run_and_check) 2026-10-29 19:43:20.423000 [ 725 ] DEBUG : Sending stat to :9181 (keeper_utils.py:271, send_4lw_cmd) 2026-10-29 19:43:20.423000 [ 725 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:20.424000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:43:20.797000 [ 725 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:43:20.802000 [ 725 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestkeeperreconfigremovemany-gw9-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:43:32.610000 [ 725 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env --project-name roottestkeeperreconfigremovemany-gw9 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.639000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:00.640000 [ 725 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.651000 [ 725 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.663000 [ 725 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.677000 [ 725 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.688000 [ 725 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:44:00.699000 [ 725 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/.env --project-name roottestkeeperreconfigremovemany-gw9 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances-0-gw9/node5/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node4-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Container roottestkeeperreconfigremovemany-gw9-node5-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Network roottestkeeperreconfigremovemany-gw9_default Removing (cluster.py:147, run_and_check) 2026-10-29 19:44:01.431000 [ 725 ] DEBUG : Stderr: Network roottestkeeperreconfigremovemany-gw9_default Removed (cluster.py:147, run_and_check) 2026-10-29 19:44:01.432000 [ 725 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:44:01.449000 [ 725 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany-gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:44:01.469000 [ 725 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:44:01.488000 [ 725 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany-gw9 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:44:01.488000 [ 725 ] DEBUG : Command:[docker container list --all --filter name='^/roottestkeeperreconfigremovemany-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:44:01.508000 [ 725 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:44:01.508000 [ 725 ] DEBUG : No running containers for project: roottestkeeperreconfigremovemany-gw9 (cluster.py:879, cleanup) 2026-10-29 19:44:01.508000 [ 725 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:44:01.524000 [ 725 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:44:01.525000 [ 725 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:44:01.543000 [ 725 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:44:01.544000 [ 725 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:44:01.544000 [ 725 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:44:01.544000 [ 725 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:44:01.564000 [ 725 ] DEBUG : Stdout:6 (cluster.py:145, run_and_check) 2026-10-29 19:44:01.564000 [ 725 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:44:01.582000 [ 725 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:44:01.583000 [ 725 ] DEBUG : Volumes pruned: 6 (cluster.py:915, cleanup) ----------------- generated report log file: parallel1_0.jsonl ----------------- ============================== slowest durations =============================== 248.94s call test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] 194.57s call test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] 165.34s call test_mysql_protocol/test.py::test_mysql_federated 124.46s call test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] 118.28s call test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] 105.80s call test_mutations_hardlinks/test.py::test_delete_and_drop_mutation 47.40s setup test_distributed_ddl/test.py::test_macro[configs] 46.83s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 46.77s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 40.49s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 37.66s call test_distributed_ddl/test.py::test_socket_timeout[configs_secure] 33.31s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 29.98s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 29.95s teardown test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 29.53s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 27.32s setup test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete 27.00s teardown test_keeper_profiler/test.py::test_profiler 26.84s call test_distributed_ddl/test.py::test_on_server_fail[configs_secure] 25.04s call test_distributed_ddl/test.py::test_socket_timeout[configs] 24.69s teardown test_log_family_s3/test.py::test_stripe_log_truncate 24.63s teardown test_move_ttl_broken_compatibility/test.py::test_bc_compatibility 24.51s call test_distributed_ddl/test.py::test_rename[configs_secure] 22.84s call test_distributed_ddl/test.py::test_simple_alters[configs] 22.80s call test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] 22.42s call test_keeper_snapshots/test.py::test_invalid_snapshot 21.77s teardown test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 21.49s setup test_mysql_protocol/test.py::test_golang_client 21.00s call test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] 20.83s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 20.46s call test_move_ttl_broken_compatibility/test.py::test_bc_compatibility 20.42s setup test_distributed_ddl/test.py::test_kill_query[configs_secure] 20.24s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 20.13s setup test_move_ttl_broken_compatibility/test.py::test_bc_compatibility 20.05s call test_distributed_ddl/test.py::test_simple_alters[configs_secure] 19.85s setup test_keeper_profiler/test.py::test_profiler 19.61s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 18.35s setup test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] 17.97s call test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] 17.38s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 17.01s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 16.98s setup test_log_family_s3/test.py::test_log_family_s3[Log] 16.54s setup test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 16.41s setup test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table 15.75s setup test_limit_materialized_view_count/test.py::test_limit_materialized_view_count 15.19s setup test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 15.15s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 14.80s setup test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 14.40s setup test_keeper_snapshots/test.py::test_ephemeral_after_restart 14.37s call test_distributed_ddl/test.py::test_rename[configs] 13.92s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 13.90s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 13.46s setup test_mutations_hardlinks/test.py::test_delete_and_drop_mutation 13.43s call test_limit_materialized_view_count/test.py::test_limit_materialized_view_count 13.04s setup test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] 13.01s call test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 13.00s setup test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant 12.70s teardown test_distributed_ddl/test.py::test_socket_timeout[configs] 12.58s call test_distributed_ddl/test.py::test_macro[configs] 12.31s call test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] 11.71s teardown test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table 10.81s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 10.70s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 10.62s teardown test_mutations_hardlinks/test.py::test_update_mutation 10.40s call test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant 10.37s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 10.19s setup test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 10.14s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 10.12s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 9.71s call test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree 9.61s call test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] 9.60s call test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 9.51s teardown test_merge_tree_empty_parts/test.py::test_empty_parts_summing 9.45s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 9.11s call test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] 9.06s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 9.03s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 8.88s call test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 8.86s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 8.74s setup test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 8.66s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 8.41s call test_mysql_protocol/test.py::test_java_client_binary 8.22s call test_distributed_ddl/test.py::test_on_server_fail[configs] 8.10s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 8.07s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 8.04s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 8.02s teardown test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 7.69s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 7.65s call test_mysql_protocol/test.py::test_java_client_text 7.50s teardown test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant 7.22s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 7.14s call test_keeper_snapshots/test.py::test_state_after_restart 7.11s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 7.01s teardown test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] 6.95s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 6.75s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 6.65s teardown test_mysql_protocol/test.py::test_types 6.64s teardown test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] 6.63s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 6.45s call test_distributed_ddl/test.py::test_replicated_without_arguments[configs] 6.24s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 6.23s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 5.89s call test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 5.75s call test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 5.75s call test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 5.58s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 5.21s call test_distributed_ddl/test.py::test_macro[configs_secure] 4.46s teardown test_limit_materialized_view_count/test.py::test_limit_materialized_view_count 4.34s call test_keeper_snapshots/test.py::test_snapshot_size 4.11s call test_log_family_s3/test.py::test_stripe_log_truncate 3.96s call test_keeper_snapshots/test.py::test_ephemeral_after_restart 3.84s call test_failed_mutations/test.py::test_exponential_backoff_setting_override 3.63s call test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table 3.48s call test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 3.44s call test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table 2.70s call test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] 2.67s call test_log_family_s3/test.py::test_log_family_s3[Log] 2.60s call test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete 2.39s call test_log_family_s3/test.py::test_log_family_s3[StripeLog] 2.31s call test_merge_tree_empty_parts/test.py::test_empty_parts_summing 2.10s call test_mutations_hardlinks/test.py::test_update_mutation 2.00s call test_log_family_s3/test.py::test_log_family_s3[TinyLog] 1.62s call test_mutations_hardlinks/test.py::test_drop_mutation 1.51s teardown test_keeper_snapshots/test.py::test_state_after_restart 1.48s call test_mutations_hardlinks/test.py::test_modify_mutation 1.31s teardown test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 0.95s call test_distributed_ddl/test.py::test_optimize_query[configs_secure] 0.80s call test_distributed_ddl/test.py::test_optimize_query[configs] 0.67s call test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 0.50s call test_mysql_protocol/test.py::test_mysql_boolean_format 0.50s call test_mysql_protocol/test.py::test_mysqljs_client 0.44s call test_mysql_protocol/test.py::test_mysql_replacement_query 0.37s call test_mysql_protocol/test.py::test_mysql_client 0.37s setup test_mysql_protocol/test.py::test_java_client_binary 0.33s setup test_mysql_protocol/test.py::test_php_client 0.32s call test_distributed_ddl/test.py::test_kill_query[configs_secure] 0.32s setup test_mysql_protocol/test.py::test_mysqljs_client 0.32s call test_keeper_profiler/test.py::test_profiler 0.29s call test_mysql_protocol/test.py::test_mysql_affected_rows 0.27s call test_mysql_protocol/test.py::test_php_client 0.24s call test_mysql_protocol/test.py::test_mysql_explain 0.16s call test_mysql_protocol/test.py::test_golang_client 0.08s call test_mysql_protocol/test.py::test_mysql_client_exception 0.07s call test_mysql_protocol/test.py::test_python_client 0.06s call test_mysql_protocol/test.py::test_mysql_select_user 0.06s call test_mysql_protocol/test.py::test_mysql_set_variables 0.03s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 0.01s call test_mysql_protocol/test.py::test_types 0.01s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 0.00s teardown test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 0.00s teardown test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] 0.00s teardown test_distributed_ddl/test.py::test_rename[configs_secure] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 0.00s teardown test_mysql_protocol/test.py::test_mysql_federated 0.00s teardown test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] 0.00s teardown test_mutations_hardlinks/test.py::test_delete_and_drop_mutation 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 0.00s setup test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s teardown test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete 0.00s setup test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 0.00s setup test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] 0.00s setup test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] 0.00s setup test_mysql_protocol/test.py::test_mysql_replacement_query 0.00s setup test_distributed_ddl/test.py::test_socket_timeout[configs_secure] 0.00s teardown test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 0.00s teardown test_distributed_ddl/test.py::test_macro[configs_secure] 0.00s teardown test_distributed_ddl/test.py::test_simple_alters[configs_secure] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 0.00s setup test_distributed_ddl/test.py::test_on_server_fail[configs_secure] 0.00s teardown test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] 0.00s setup test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 0.00s setup test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table 0.00s setup test_mutations_hardlinks/test.py::test_drop_mutation 0.00s teardown test_distributed_ddl/test.py::test_optimize_query[configs_secure] 0.00s setup test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] 0.00s setup test_mysql_protocol/test.py::test_java_client_text 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 0.00s teardown test_distributed_ddl/test.py::test_macro[configs] 0.00s teardown test_keeper_snapshots/test.py::test_invalid_snapshot 0.00s setup test_keeper_snapshots/test.py::test_snapshot_size 0.00s setup test_distributed_ddl/test.py::test_optimize_query[configs_secure] 0.00s teardown test_distributed_ddl/test.py::test_kill_query[configs_secure] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 0.00s teardown test_mysql_protocol/test.py::test_java_client_binary 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 0.00s setup test_keeper_snapshots/test.py::test_state_after_restart 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s teardown test_keeper_snapshots/test.py::test_snapshot_size 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 0.00s teardown test_keeper_snapshots/test.py::test_ephemeral_after_restart 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 0.00s setup test_merge_tree_empty_parts/test.py::test_empty_parts_summing 0.00s teardown test_mutations_hardlinks/test.py::test_modify_mutation 0.00s setup test_distributed_ddl/test.py::test_optimize_query[configs] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] 0.00s teardown test_mysql_protocol/test.py::test_mysql_affected_rows 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 0.00s setup test_distributed_ddl/test.py::test_on_server_fail[configs] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 0.00s setup test_distributed_ddl/test.py::test_replicated_without_arguments[configs] 0.00s teardown test_distributed_ddl/test.py::test_socket_timeout[configs_secure] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 0.00s teardown test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 0.00s setup test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 0.00s setup test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 0.00s teardown test_distributed_ddl/test.py::test_on_server_fail[configs_secure] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 0.00s setup test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 0.00s teardown test_failed_mutations/test.py::test_exponential_backoff_setting_override 0.00s setup test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_distributed_ddl/test.py::test_rename[configs_secure] 0.00s setup test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] 0.00s setup test_mysql_protocol/test.py::test_types 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 0.00s setup test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_distributed_ddl/test.py::test_rename[configs] 0.00s teardown test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree 0.00s teardown test_mysql_protocol/test.py::test_java_client_text 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 0.00s teardown test_distributed_ddl/test.py::test_on_server_fail[configs] 0.00s teardown test_distributed_ddl/test.py::test_rename[configs] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 0.00s setup test_mutations_hardlinks/test.py::test_update_mutation 0.00s teardown test_mysql_protocol/test.py::test_mysql_client_exception 0.00s teardown test_mysql_protocol/test.py::test_python_client 0.00s teardown test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] 0.00s setup test_keeper_snapshots/test.py::test_invalid_snapshot 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 0.00s setup test_distributed_ddl/test.py::test_macro[configs_secure] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_failed_mutations/test.py::test_exponential_backoff_setting_override 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_mysql_protocol/test.py::test_mysql_affected_rows 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 0.00s setup test_mysql_protocol/test.py::test_mysql_set_variables 0.00s setup test_distributed_ddl/test.py::test_simple_alters[configs] 0.00s teardown test_mysql_protocol/test.py::test_golang_client 0.00s setup test_mysql_protocol/test.py::test_mysql_boolean_format 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] 0.00s setup test_mysql_protocol/test.py::test_mysql_explain 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 0.00s setup test_mysql_protocol/test.py::test_mysql_select_user 0.00s setup test_distributed_ddl/test.py::test_socket_timeout[configs] 0.00s setup test_mutations_hardlinks/test.py::test_modify_mutation 0.00s setup test_distributed_ddl/test.py::test_simple_alters[configs_secure] 0.00s setup test_mysql_protocol/test.py::test_mysql_federated 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_log_family_s3/test.py::test_stripe_log_truncate 0.00s setup test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree 0.00s teardown test_mysql_protocol/test.py::test_mysql_replacement_query 0.00s teardown test_mutations_hardlinks/test.py::test_drop_mutation 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 0.00s setup test_mysql_protocol/test.py::test_python_client 0.00s setup test_mysql_protocol/test.py::test_mysql_client_exception 0.00s teardown test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] 0.00s teardown test_mysql_protocol/test.py::test_mysqljs_client 0.00s teardown test_mysql_protocol/test.py::test_mysql_select_user 0.00s teardown test_mysql_protocol/test.py::test_php_client 0.00s teardown test_distributed_ddl/test.py::test_replicated_without_arguments[configs] 0.00s setup test_mysql_protocol/test.py::test_mysql_client 0.00s teardown test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] 0.00s teardown test_mysql_protocol/test.py::test_mysql_explain 0.00s teardown test_distributed_ddl/test.py::test_optimize_query[configs] 0.00s teardown test_mysql_protocol/test.py::test_mysql_set_variables 0.00s teardown test_mysql_protocol/test.py::test_mysql_client 0.00s teardown test_distributed_ddl/test.py::test_simple_alters[configs] 0.00s teardown test_mysql_protocol/test.py::test_mysql_boolean_format =========================== short test summary info ============================ FAILED test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader PASSED test_keeper_snapshots/test.py::test_ephemeral_after_restart PASSED test_log_family_s3/test.py::test_log_family_s3[Log] PASSED test_distributed_ddl/test.py::test_kill_query[configs_secure] PASSED test_mysql_protocol/test.py::test_golang_client PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] PASSED test_distributed_ddl/test.py::test_macro[configs_secure] PASSED test_log_family_s3/test.py::test_stripe_log_truncate PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] PASSED test_mysql_protocol/test.py::test_java_client_binary PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] PASSED test_mysql_protocol/test.py::test_java_client_text PASSED test_mysql_protocol/test.py::test_mysql_affected_rows PASSED test_mysql_protocol/test.py::test_mysql_boolean_format PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] PASSED test_mysql_protocol/test.py::test_mysql_client PASSED test_mysql_protocol/test.py::test_mysql_client_exception PASSED test_mysql_protocol/test.py::test_mysql_explain PASSED test_keeper_snapshots/test.py::test_invalid_snapshot PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] PASSED test_failed_mutations/test.py::test_backoff_clickhouse_restart[False] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] PASSED test_keeper_snapshots/test.py::test_snapshot_size PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] PASSED test_keeper_snapshots/test.py::test_state_after_restart PASSED test_distributed_ddl/test.py::test_on_server_fail[configs_secure] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] PASSED test_distributed_ddl/test.py::test_optimize_query[configs_secure] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] PASSED test_failed_mutations/test.py::test_backoff_clickhouse_restart[True] PASSED test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table PASSED test_failed_mutations/test.py::test_exponential_backoff_setting_override PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] PASSED test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant PASSED test_distributed_ddl/test.py::test_rename[configs_secure] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True] PASSED test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] PASSED test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] PASSED test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table PASSED test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete PASSED test_distributed_ddl/test.py::test_simple_alters[configs_secure] PASSED test_merge_tree_empty_parts/test.py::test_empty_parts_summing PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] PASSED test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] PASSED test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] PASSED test_keeper_profiler/test.py::test_profiler PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] PASSED test_distributed_ddl/test.py::test_socket_timeout[configs_secure] PASSED test_limit_materialized_view_count/test.py::test_limit_materialized_view_count PASSED test_mutations_hardlinks/test.py::test_delete_and_drop_mutation PASSED test_mutations_hardlinks/test.py::test_drop_mutation PASSED test_mutations_hardlinks/test.py::test_modify_mutation PASSED test_mutations_hardlinks/test.py::test_update_mutation PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery PASSED test_move_ttl_broken_compatibility/test.py::test_bc_compatibility PASSED test_mysql_protocol/test.py::test_mysql_federated PASSED test_mysql_protocol/test.py::test_mysql_replacement_query PASSED test_mysql_protocol/test.py::test_mysql_select_user PASSED test_mysql_protocol/test.py::test_mysql_set_variables PASSED test_distributed_ddl/test.py::test_macro[configs] PASSED test_mysql_protocol/test.py::test_mysqljs_client PASSED test_mysql_protocol/test.py::test_php_client PASSED test_mysql_protocol/test.py::test_python_client PASSED test_mysql_protocol/test.py::test_types PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] PASSED test_distributed_ddl/test.py::test_on_server_fail[configs] PASSED test_distributed_ddl/test.py::test_optimize_query[configs] PASSED test_distributed_ddl/test.py::test_rename[configs] PASSED test_distributed_ddl/test.py::test_replicated_without_arguments[configs] PASSED test_distributed_ddl/test.py::test_simple_alters[configs] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server] PASSED test_distributed_ddl/test.py::test_socket_timeout[configs] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server] PASSED test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user] SKIPPED [1] test_library_bridge/test_exiled.py:54: Leak sanitizer falsely reports about a leak of 16 bytes in clickhouse-odbc-bridge ============= 1 failed, 98 passed, 1 skipped in 708.00s (0:11:47) ============== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_m6exb1 --privileged --dns-search='.' --memory=30703235072 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel1_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_distributed_ddl/test.py::test_kill_query[configs_secure]' 'test_distributed_ddl/test.py::test_macro[configs]' 'test_distributed_ddl/test.py::test_macro[configs_secure]' 'test_distributed_ddl/test.py::test_on_server_fail[configs]' 'test_distributed_ddl/test.py::test_on_server_fail[configs_secure]' 'test_distributed_ddl/test.py::test_optimize_query[configs]' 'test_distributed_ddl/test.py::test_optimize_query[configs_secure]' 'test_distributed_ddl/test.py::test_rename[configs]' 'test_distributed_ddl/test.py::test_rename[configs_secure]' 'test_distributed_ddl/test.py::test_replicated_without_arguments[configs]' 'test_distributed_ddl/test.py::test_replicated_without_arguments[configs_secure]' 'test_distributed_ddl/test.py::test_simple_alters[configs]' 'test_distributed_ddl/test.py::test_simple_alters[configs_secure]' 'test_distributed_ddl/test.py::test_socket_timeout[configs]' 'test_distributed_ddl/test.py::test_socket_timeout[configs_secure]' 'test_failed_mutations/test.py::test_backoff_clickhouse_restart[False]' 'test_failed_mutations/test.py::test_backoff_clickhouse_restart[True]' test_failed_mutations/test.py::test_exponential_backoff_create_dependent_table test_failed_mutations/test.py::test_exponential_backoff_setting_override 'test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node0-True]' 'test_failed_mutations/test.py::test_exponential_backoff_with_merge_tree[node1-False]' test_failed_mutations/test.py::test_exponential_backoff_with_replicated_tree 'test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[False]' 'test_failed_mutations/test.py::test_no_backoff_after_killing_mutation[True]' test_grant_and_revoke/test_without_table_engine_grant.py::test_table_engine_and_source_grant test_groupBitmapAnd_on_distributed/test_groupBitmapAndState_on_distributed_table.py::test_groupBitmapAndState_on_distributed_table test_keeper_profiler/test.py::test_profiler test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_snapshots/test.py::test_ephemeral_after_restart test_keeper_snapshots/test.py::test_invalid_snapshot test_keeper_snapshots/test.py::test_snapshot_size test_keeper_snapshots/test.py::test_state_after_restart test_library_bridge/test_exiled.py::test_bridge_dies_with_parent test_limit_materialized_view_count/test.py::test_limit_materialized_view_count 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_log_family_s3/test.py::test_stripe_log_truncate test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[server]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_group_by[user]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[server]' 'test_max_bytes_ratio_before_external_order_group_by_for_server/test.py::test_max_bytes_ratio_before_external_sort[user]' test_merge_tree_empty_parts/test.py::test_empty_parts_alter_delete test_merge_tree_empty_parts/test.py::test_empty_parts_summing 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_move_ttl_broken_compatibility/test.py::test_bc_compatibility test_mutations_hardlinks/test.py::test_delete_and_drop_mutation test_mutations_hardlinks/test.py::test_drop_mutation test_mutations_hardlinks/test.py::test_modify_mutation test_mutations_hardlinks/test.py::test_update_mutation test_mysql_protocol/test.py::test_golang_client test_mysql_protocol/test.py::test_java_client_binary test_mysql_protocol/test.py::test_java_client_text test_mysql_protocol/test.py::test_mysql_affected_rows test_mysql_protocol/test.py::test_mysql_boolean_format test_mysql_protocol/test.py::test_mysql_client test_mysql_protocol/test.py::test_mysql_client_exception test_mysql_protocol/test.py::test_mysql_explain test_mysql_protocol/test.py::test_mysql_federated test_mysql_protocol/test.py::test_mysql_replacement_query test_mysql_protocol/test.py::test_mysql_select_user test_mysql_protocol/test.py::test_mysql_set_variables test_mysql_protocol/test.py::test_mysqljs_client test_mysql_protocol/test.py::test_php_client test_mysql_protocol/test.py::test_python_client test_mysql_protocol/test.py::test_types 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.